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ABSTRACT 



The GALAXY Classroom is a package of integrated curricular 



and instructional approaches, supported by the first U.S. interactive 
satellite communications network designed to facilitate the introduction of 
innovative curricula to improve student learning in elementary schools. 

GALAXY Classroom Science for grades 3-5 features the organization of 
instruction around themes presented through television broadcasts and 
classroom hands-on activities that are facilitated by fax technology and 
ongoing teacher support. The broadcasts are built around a video adventure 
series that dramatizes the GALAXY themes through the real-world adventures of 
a multicultural group of students, the S.N.O.O.P.S. The report describes the 
effects on students and teachers of the various curricular, technological, 
and staff development components of the program in schools throughout the 
United States. It was concluded that the GALAXY Science program was 
successful in getting students and teachers excited about science and 
motivating them to begin exploring and asking questions. GALAXY'S broadcasts 
motivated student interest and provided a realistic context for the hands-on 
science activities. The thematic structure of the curriculumi helped to 
integrate a wide variety of experiences: participating in hands-on inquiry, 
writing and communicating through the fax, and doing take-home activities. 
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I. INTRODUCTION 



"GALAXY has enriched my teaching. It has made me enthusiastic and happy 
about teaching science. I have enjoyed and learned a lot about how children 
learn, i.e., investigate, explore, share. Having the materials readily available 
has been GREAT. My teaching has been RECHARGED!!!" 

— GALAXY teacher from California 

The Galaxy Classroom is a package of integrated curricular and instructional approaches, 
supported by the nation's first interactive satellite corrununications network designed to 
facilitate the introduction of innovative curricula to improve student learning in elementary 
schools. Challenging curricula and learning experiences are made available to all students, 
including those who ordinarily lack such access. 

GALAXY Classroom Science for Grades 3-5 features the organization of instruction around 
themes presented through television broadcasts and classroom hands-on activities that are 
facilitated by fax technology and ongoing teacher support. The broadcasts are built around a 
video adventure series that dramatizes the GALAXY themes through the recd-world adventures 
of a multicultural group of students, THE S.N.O.O.P.S. The stories feature characters with 
whom students can identify and who model for students the use of various scientific techniques 
and processes to explore and understand their world. 

The evaluation found that GALAXY science for grades 3-5 is a highly successful initiative: 

• On measures of classification processes, GALAXY students had a statistically 
significant gain that was more tiian double the gain of non-GALAXY comparison 
students. 

• Scores on curriculiun-based performance assessments indicate that the majority of 
GALAXY students across all three grades were able to demonstrate that they 
understood the "big ideas" or core science concepts of the GALAXY curriculum. 

• In genercd, when comparison non-GALAXY students were evaluated on some of the 
same measures, GALAXY students outperformed them in almost every case. 

• In addition, GALAXY teachers displayed significantly more positive attitudes than they 
had initially regarding their own comfort with and preparation for teaching science and 
the adequacy of their science materials. 

• Participating in GALAXY Classroom Science led to statistically significant positive 
changes in attitudes among GALAXY students, when compared to their non-GALAXY 
peers, toward participating in science class and engaging in activities to which they did 
not know the right answer. 

Evidence from the evcduation shows that GALAXY works: in GALAXY classrooms science is 
being regularly taught, students are learning, and teachers are developing a new enthusiasm for 
science and science teaching. The evcduation further shows that the whole of GALAXY science 
is greater than the sum of its parts. The components by themselves represent exemplary efforts 
in science education. Together, they create a powerful package. 



Executive Summary of Evaluation Final Report 
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II. Evaluation Overview 



"We get to touch things and discover stuff. " 

— GALAXY student 

The GALAXY Classroom Project was developed as a nationwide reform effort to infuse new 
student-centered curricula into elementary schools, to spark the interest of teachers and 
students in learning, and to provide innovative educational opportunities for all students 
including those who have traditionally been considered academically "at risk." 

GALAXY Classroom Science for Grades 3-5 

The curriculum for GALAXY Classroom Science for Grades 3-5 is built upon the fundamental 
view that students can construct knowledge about science from the content and context of their 
daily lives. GALAXY'S goals for elementary science have been three: to foster the development 
of the thinking processes that are tools for constructing knowledge of science (e.g., observing, 
communicating, comparing, organizing, relating), to provide students with some of the 'big 
ideas" or core science concepts that are fundamental to understanding science, and to aid 
teachers through ongoing professional support. 

"Although I am a veteran teacher, I felt at a loss as to how to teach science. The 
concepts of the arcs [themes] — patterns, experiments, and black boxes — made sense to 
me.... I could teach the concepts rather than information and knowledge I frequently 
lack. " 

— GALAXY teacher from California 

The 'big ideas" are communicated through the GALAXY inquiry-based, hands-on/minds-on 
science curriculum which is organized around three six-week themes: (1) using patterns as 
evidence, (2) doing experiments to describe and compare materials, and (3) builchng models to 
explain and invent ideas. Each of these themes was designed to be age-appropriate; to use 
investigations that teachers can expand; to work in an interdisciplinary science program; and to 
support national, state, and local frameworks and standards. 

The Evaluation Approach 

The Far West Laboratory for Educational Research and Development (FWL) conducted a 
comprehensive evaluation of the implementation and impact of GALAXY Classroom Science for 
Grades 3-5 for the initial demonstration phase. 

The evaluation gathered quantitative data on GALAXY'S impact by testing student learning 
throu^ performance-based assessments, surveying student and teacher attitudes and teacher 
practices, and asking teachers to record their use of the GALAXY Classroom Science curriculum. 
Performance-based assessments, in contrast to most multiple-choice tests, ask students to do 
something and then record what they have discovered. Administration of four of the 
performance-based assessments and the attitude surveys followed a pre/post design. Four 
other assessments were more closely linked to the curriculum and activities, and they were 
administered during the course of GALAXY science. 

In addition to these quantitative measures, a series of observations and interviews were carried 
out in five case study schools throughout the coxmtry, with shorter visits to several other 
schools. These case study classrooms were spread across the nation and varied in the ethnicity 
of the students as well as whether their setting was urban, suburban, or rural. 
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GALAXY Classroom Science for Grades 3-5 Demonstration Program 

GALAXY Classroom Science for Grades 3-5 is a package of integrated curricular themes and 
hands-on science activities. Instruction is organized around three themes presented through 
television broadcasts and classroom hands-on science activities and supported by the use of 
fax technology in the classroom and take-home science activities. 

During the demonstration phase for GALAXY Classroom Science for Grades 3-5, classrooms in 
thirty-eight schools (later forty) were connected by an interactive satellite communications 
network. The demonstration phase ran for eighteen weeks of class time in the fall of 1993 and 
winter of 1994. 

"[GALAXY] worked well with all students because of high interest cliff-hanger 
broadcasts backed up with interesting, program-coordinated hands-on learning 
activities. " 

— GALAXY teacher from West Virginia 

Broadcasts The television broadcasts dramatized the GALAXY themes through a continuing 
program about the adventures of a multicultural group of students, THE S.N.O.O.P.S., 
interacting together as they solved various mysteries. The stories featured characters with 
whom students could identify and modeled for students the use of various scientific techniques 
that the characters employed to solve the problems and puzzles they encountered. 

"The program hooks in all kids because they can relate to it. " 

— GALAXY teacher from California 

"They put your name on TV! We can't wait 'til the program comes on 
again. " 

— GALAXY student 

The evaluation found that students were entranced with the broadcasts; the broadcast 
mysteries received the highest ratings from students at case study schools among all the 
GALAXY components about which they were asked. Similarly, more than 90% of the teachers 
who completed the end-of-year survey judged the broadcasts to be of "great educational value." 

Hands-on Activities Each classroom used a set of GALAXY-provided hands-on science kits 
that are parts of the Great Explorations in Math and Science (GEMS) or the Full Option Science 
System (FOSS) curricula, both developed by the Lawrence Hall of Science at the University of 
California, Berkeley. 

"Hands-on activities are wonderful. These activities really reach the interest levels of all 
students. " 

— GALAXY teacher from West Virginia 

FWL evaluators found that both teachers and students were extremely pleased with the 
GALAXY hands-on materials and activities. Teacher enthusiasm at case study sites stemmed 
from having all the materials they needed and receiving them at one time, using the GEMS and 
FOSS guides that were easy to follow and adapt for their classes, and knowing that students 
loved the activities and were therefore eager and interested learners. This enthusiasm was 
echoed in the end-of-year survey when nearly all the teachers rated the activities as having 
"great educational value." 

"Hands-on activities and take-home kits made the kids feel like they could do anything. 

Executive Summary of Evaluation Final Report 
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— GALAXY teacher from Indiana 



Teacher enthusiasm was tempered in a few cases by factors such as the significant amount of 
time necessary for setup, the inevitable mess associated with hands-on materials, and the 
perception among some teachers that a few of the activities were too difficult for third graders. 
In addition, some teachers thought that the Teacher's Guide could have provided more 
assistance on how to link the ideas and themes of the broadcast with the hands-on experiences. 

Students were, if anything, even more enthusiastic than their teachers about the hands-on 
activities. More than 90% of the interviewed students gave the activities their highest rating. 

Fax One of GALAXY'S unique features is facilitating student and teacher interaction with one 
another across the city or across the country and with the broadcasts' producers through a 
dedicated satellite network and classroom fax machines. 

"I think that GALAXY Science worked well with all of my students because their ideas 
were recognized no matter what they submitted (i.e., pictures, photographs, etc.). 

— GALAXY teacher from Massachusetts 

Interviewed students were very positive about faxing, some citing the chance of seeing their 
names in the broadcast. Additionally, many of the students' ideas that were sent to Database 
(headquarters for THE S.N.O.O.P.S.) appeared in The Scoop, a two-page fax bulletin that was 
sent periodically to GALAXY classrooms. However, fax problems prevented some classes from 
receiving it. 

Teachers also valued faxing, with more than 80% of teachers who responded to the end-of-year 
survey rating sending faxes as having "great educational value" for their students and more than 
70% reporting that students were "very enthusiastic" about sending faxes to Database. An even 
higher proportion of teachers said that their students were "very enthusiastic" about seeing their 
names and/or faxes on the broadcasts. 

"The fax machine is integral. The kids were disappointed when they worked for days 
on faxes to S.N.O.O.P.S. and missed the fax deadline because our fax machine did not 
work. " 

— GALAXY teacher from Wisconsin 

During the demonstration phase, technical difficulties produced frustration for many teachers. 
Only 12% reported that their fax machine always worked, and others commented that they 
were never sure whether their faxes reached their destinations. When these kinds of technical 
problems occurred, whether at the classroom, school, or network level, students became 
disappointed: "Our faxes never got to Database — they got ate up by the satellite." 

FWL evaluators concluded that, when the fax machines worked, they provided powerful 
motivational incentives for GALAXY students. 

Snoopers Loop Take-Home Kits Materials for each theme included four different home 
investigation science kits that students were encouraged to explore with family members and 
friends. All the necessary materials plus instructions for doing an investigation at home with an 
adult were included in each kit. 

"My class begged for the take-homes. Please keep them in. " 

— GALAXY teacher from Pennsylvania 
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Each of the take-home kits arrived at school in parts and required time for assembly (e.g., 
placing each of the parts and instructions in plastic bags). While some teachers asked parents 
to help put together the kits or held a "family science night," others were overwhelmed by trying 
to do it themselves. However, even though preparing the kits was fairly labor-intensive, they 
were rated well by both students interviewed (more than 70% at case study sites) and the 
majority of teachers (more than 60% rated them as "very useful" and having "great educational 
value"). 
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III. Developing Scientific Thinking 
Processes and Results from 
Performance-Based Assessments 



Recognizing that there is a progression to the development of scientific thinking skills, 
researchers at FWL adapted four performance-based assessments from the California Learning 
Assessment System (CLAS) to test GALAXY and comparison students' progress in several 
crucial areas. FWL researchers measured classification and organization (critical components 
of scientific thinking for grades 3-4 and 5-6) with two hands-on assessments using fossils in the 
pre-test and leaves in the post-test. Skills related to experimentation were measured by two 
other pre/post performance-based assessments using rocks and soils, which were administered 
in a crossover design. Additionally, students took a multiple-choice test of science process 
skills. 

Results from the Classification Pre/Post Assessments 

The evidence shows that participation in GALAXY had a statistically significant positive effect 
on students' classification abilities. These results are based on testing 600 GALAXY and 610 
comparison students in the same grades at twelve GALAXY schools. Each of the two 
assessments had three tasks that were scored from 0 (no attempt) to 5 (accurate and 
informative). Figure 1 shows the average (mean) scores for GALAXY and comparison students 
in each of the three grades, both before GALAXY science started (pre) and after it was 
completed (post). 




Figure 1. Pre/Post Mean Total Scores on Classification Tasks 

• GALAXY students had statistically significantly greater gains than comparison 
students on each of the three classification tasks in the assessments. (Task 1: .50 vs. 
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.19, p<.001; Task 2: .34 vs. .11, p=.016; Task 3: .79 vs. .48, p<.001). 

• GALAXY students had statistically significantly greater gains than comparison 
students at each grade level (third, fourth, and fifth). 

• Figure 1 illustrates the GALAXY and comparison mean gain at each grade for overall 
classification scores (the sum of the three individual task scores). The most dramatic 
result is that, for third and fourth grades, GALAXY student post-test scores surpassed not 
only those of their grade level counterparts but also the pre-test and post-test scores of 
comparison students in the next grade. 

• Looking at all the data from all three grades together, students who participated in 
GALAXY science had a statistically significant gain in classification abilities that was 
more than double that of comparison students (1.63 vs. .79, p<.001). 

• The gams measured by this assessment were similar across all three grade levels and 
unaffected by gender, ethnicity. Chapter 1 status, language spoken at home, or previous 
participation in GALAXY Language Arts. 

• GALAXY is appropriate for both high-achieving and academically at-risk students as 
evidenced by fairly similar gains for students regardless of their initial performance on 
these assessments. 

Observing, communicating, comparing, and classifying are crucial scientific thinking processes 
for students in grades 3-5. Based on the results of the classification assessments, GALAXY 
appears to have met the goal of fostering the development of these processes. Indeed, the results 
suggest that GALAXY science can work for every type of student and can carry many of them up to a 
par with students who are a year ahead of them. 

Results From Pre/Post Assessments of Experimental Skills 

During the fifth and sixth grades, students expand their repertoire of scientific thinking skills to 
include the abilities to recognize relationships between ideas and to design simple experiments. 
Ehiiing this period students also develop ^e ability to isolate and manipulate variables in an 
increasingly systematic manner. 

Researchers at FWL tested these experimental skills by adapting two existing CLAS 
performance-based assessments to use in a pre/post crossover evaluation design. GALAXY 
and comparison students participated in either the Rocks or the Soils assessment as a pre-test 
and the other as a post-test. Unlike the classification assessments, which yielded results for 
individual students, the assessments of experimental skills gave information about groups of 
students. Because the skills being measured were known to be developmentally beyond most 
third graders, the tests were administered to fourth and fifth graders only. 

Results from the Rocks Assessment 

The patterns of scores shown by fourth and fifth graders on the four tasks of this assessment 
reflect the difference in their developmental levels of imderstanding about experimentation. 
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Results for fourth graders indicate no pattern of significant change on this test. Although some 
growth is evident, neither comparison nor GALAXY fourth graders show gains that are 
statistically significant. 

In contrast to the results for fourth graders, an interesting and promising pattern emerges for 
GALAXY fifth graders on the essential components of the test, particularly in independently 
repeating experiments (Task 2) and in using evidence to support experimental conclusions 
(Task 4). 

• On Tasks 2 and 4 of this assessment, the post-test GALAXY fifth grade cohort shows 
statistically significantly higher scores when compared to pre-test GALAXY students. 

• GALAXY fifth graders performed statistically significantly better than comparison 
students on Task 2, and, although GALAXY students display greater gains than 
comparison students on Task 4, it falls short of statistical significance at the .05 level 
(p=.084). 

• Results from the other two tasks indicate no real difference between the GALAXY and 
comparison fifth graders. 

Although the Rocks assessment did not prove to be an age-appropriate measure for fourth 
graders, several of its components did help to demonstrate that fifth graders who participated in 
GALAXY Science were better able than their non-GALAXY peers to recognize relationships, isolate and 
manipulate variables in an experiment, and generalize information to new situations. 

Results from the Soils Assessment 

The Soils assessment, like the Rocks assessment, tested more advanced scientific thinking 
processes: observing, understanding relationships, and making simple inferences and 

predictions. Similar to findings from the Rocks assessment, the developmental differences 
between fourth and fifth graders appear to be reflected in the results. 

The fourth grade results for the Soils assessment show little difference in performance between 
GALAXY and comparison students. The results for the fifth graders indicate both that the test 
was more appropriate for these students and that GALAXY students outperformed their 
comparison counterparts: 

• The vast majority of GALAXY and comparison fifth graders got good scores on Task 1 
for both pre-tests and post-tests. It would appear that this measures something that 
they already know (how to replicate an experiment and observe accurately). 

• Task 2 shows a statistically significant positive change for GALAXY fifth graders and a 
smaller, not statistically significant, change for comparison students, but the difference 
in their growth was not statistically significant. 

• On Task 3, GALAXY fifth graders showed a statistically significant gain between the 
two administrations (from 1.24 to 1.74, a gain of .50). The two groups of comparison 
students scored roughly the same at pre-test and post- test (1.43 and 1.40), indicating 
that they had not learned much in this area. The difference in gain is statistically 
significant (p=.01). These results suggest that students participating in GALAXY had 
learned more than comparison fifth graders about generalizing information they had 
just learned to a new situation. 

The fifth grade results from the Soils assessment and, in particular. Task 3, indicate that GALAXY 
Classroom Science helps foster the development of scientific thinking processes, in particular comparing 
variables, relating procedures, and solving new problems. 
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Results from the Science Process Assessment 

FWL administered a multiple-choice test of scientific thinking processes, the Science Process 
Assessment, developed in ^e late 1980's to reflect curricular reform in Pennsylvania. This test 
was selected to assess scientific thinking in areas not specifically addressed in the GALAXY 
curriculum. Figure 2 shows the results in terms of mean total scores. 

The results are ambiguous for the fifth grade, but they are more clear for the fourth grade. 

• The data suggests that students in fourth grade GALAXY classrooms learned an array 
of scientific reasoning skills reflective of the new reforms in science education 
(particularly forming hypotheses and experimenting) better than their peers in 
comparison classrooms. 

• GALAXY fourth graders had gains on the thirty-one item test that were statistically 
significantly greater than those of comparison students (1.93 and .05, respectively, 

p<.001). 






— GALAXY5 




* ' Comparison 5 




-GALAXY4 


....... 


‘ ‘ Comparison 4 



Figure 2. Mean Total Scores on Modified Science Process Assessment for 
GALAXY and Comparison Students 



These results, comparing GALAXY and non-GALAXY students, suggest that GALAXY science 
can contribute to the development of scientific thinking processes among fourth graders. 
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TV . Understanding the Big Ideas of 
THE Three GALAXY Themes and 
Results from Performance-Based 

Assessments 



FWL evaluators devised a series of four curriculum-embedded performance assessments to 
evaluate students' vmderstanding of the "big ideas" presented in the three themes of the 
demonstration phase curriculum. GALAXY classrooms were asked to participate in all four 
embedded assessments; comparison students participated in two. In addition, small groups of 
students were videotaped while working on tasks related to the core science concepts of the 
three themes. 

Results for Theme 1 : Science is Finding Patterns as Evidence 

The GALAXY curriculum for Theme 1 had students viewing broadcasts and participating in 
activities that focus on recognizing and using patterns as evidence to solve crimes or explain 
events. Results from both the Theme 1 videotaped performance assessment and the 
performance-based embedded assessment {Reading the Beach) show that a large majority of 
third, fourth, and fifth grade GALAXY students understood and could apply what they had 
been taught about finding and using patterns as evidence. . 




Figure 3. Reading the Beach Scores for Comparison and GALAXY Students 
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• The overall difference in the distributions of scores between GALAXY students and non- 
GALAXY comparison students, as shown in Figure 3, is statistically significant (chi- 
square=17.98 with 5 d.f., p=.003) and indicates that GALAXY students outperformed 
their comparison peers on this assessment. 

• As Figure 3 shows, 75.5% of GALAXY students demonstrated a level of competence or 
better (score of 3, 4, or 5 out of 5) compared with 68.6% of comparison students. 

On the embedded assessment for Theme 1, in which comparison students also participated, 
GALAXY students demonstrated that they were more skilled at recognizing, using, and interpreting 
patterns than their non-GALAXY peers. 

Results for Theme 2: Science is Doing Experiments 

Theme 2 focused on the core concept of doing experiments to describe and compare materials. 
GALAXY Classroom students joined the broadcast characters in trying to solve mysteries 
through hands-on experiihentation with familiar materials. The purpose of the FWL 
videotaped assessment in this theme was twofold: to see if GALAXY facilitated students' 
working in small cooperative groups and to determine if GALAXY students approached the 
open exploration of a novel substance with greater curiosity or in a more systematic manner 
than non-GALAXY students. 

• More than half of the GALAXY groups were able to organize themselves and to assign 
themselves different roles while none of the non-GALAXY groups organized themselves 
for the task. 

• About half of the GALAXY groups, but only one of the non-GALAXY groups, 
systematically tested the properties of the unknown substances, spontaneously making 
predictions and then revising them based upon testing. 

These results from case study schools indicate that GALAXY students understood the core 
concepts of Theme 2 and were better at working in small groups and at systematic testing. 

The embedded performance assessment for Theme 2, Chemical Reactions, asked GALAXY 
students to (1) observe an experiment involving chemicals, (2) change the variables in some 
way, (3) perform the revised experiment and record observations, (4) draw a conclusion based 
upon the two experiments, and (5) design another experiment using the same chemicals and 
make a prediction about its results. 
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Figure 4. Percent of GALAXY Students Achieving Competency on 
Chemical Reactions Tasks 

• As Figure 4 shows, about 71% of the GALAXY students consistently demonstrated on 
three or more of the five parts of the assessment that they understood the concept of 
experimentation and could manipulate variables and predict what would happen. 

• It is not surprising that the students had difficulty explicitly stating a cause and effect 
relationship among the variables (part 4) because of the complexity of the thinking 
involved. 

These results indicate that GALAXY students had developed an implicit understanding of the concepts 
involxxd in experimentation, but many were not yet able to reflect upon that ur^erstanding and 
communicate it explicitly to others. 

Results for Theme 3: Science is Building Models 

The "big idea" presented in Theme 3 involved building and using models to invent and explain 
ideas. While watching the broadcasts, GALAXY students were challenged to explain surprising 
phenomena in a "haunted" theater. In addition, GALAXY gave students hands-on practice in 
the classroom with building and using a variety of models. 

The videotaped performance assessment for this theme asked students working in pairs to 
construct a model from materials provided. The model was to be a device that would solve a 
specific problem (retrieve a cat stuck in a tree). The results for this assessment show no 
difference between GALAXY and non-GALAXY students. Possible explanations for this finding 
include the lack of time spent on this theme because of inclement weather during the winter and 
a perception by some teachers that a few of the Theme 3 activities were too difficult for their 
students. 

The curriculum-embedded assessment for Theme 3, Models and Designs, asked students to use 
models in three different ways. Students were asked to design a model that was based on the 
redesign of an existing object (in this case, a bicycle), a model of what was inside a black box (a 
drinking fountain), and lastly, a model of a tool that could be used to solve a specific problem 
(unstick a basketball stuck in a net). In each case students were asked to draw a model and to 
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write a description explaining their model. The results show that GALAXY students performed 
very well on the three tasks of this assessment; the assessment was not given to comparison 
students. 
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Figure 5. Percent of GALAXY Students Achieving Competency on Models 

and Designs Tasks 

• For GALAXY students at all grade levels, the percentage of students demonstrating 
competence or above were 65.3%, 88.3%, and 87.9% respectively for the three tasks. 

• Figure 5 gives the distribution of competency among GALAXY students who 
participated in Models and Designs and shows that, under fairly stringent criteria, two- 
thirds of the GALAXY students across all grades achieved competency on two or more 
of three tasks, and 39% could display competency on all three tasks. 

Although each task tested a different aspect of using models, the evidence from this assessment 
suggests that students displayed competence at using models generally. 

Culminating Embedded Assessment: A Mystery 

A Mystery, the final embedded assessment, was given to both GALAXY and comparison 
students after GALAXY science was completed. It was intended to evaluate the ways that 
students reason about and investigate the possible causes of unexplained phenomena (strange 
noises being emitted from an abandoned house). Students were asked to choose among three 
possible explanations for the cause of the noise — a ghost, people playing tricks, or the wind. 
"Wind" and "people" were considered appropriate responses; "ghost" indicated that the 
student might still rely on magical explanations. In addition, students were asked to describe 
what they could do to determine if their explanation was correct. 
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Figure 6. Percentage of Students Choosing Each Explanation for Task 1 of A Mystery by 

Grade for All GALAXY and Comparison 

• As Figure 6 shows, GALAXY students more consistently chose "the wind" or "people" 
over "ghosts" as a plausible explanation for the noises than did their non-GALAXY 
peers. 

• Significantly more GALAXY students (29.4%) chose experimental approaches for 
deterrriirung the real cause of the noise over simple observation or explanation than did 
comparison students (20.3%), as Figure 7 shows. 

On both parts of this assessment, GALAXY students outperformed their comparison peers at 
levels that are statistically significant. These results indicate that GALAXY science was 
successful in achieving its goal of reducing magical thinking (fewer GALAXY students chose 
"ghost") and increasing understanding of experimentation (more GALAXY students chose 
active or inactive testing). 
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Figure 7. Scores of Matched GALAXY and Comparison Students on Task 2 of A 

Mystery 



Summary of Embedded Assessments 

The special videotaped performance assessments and performance-based embedded 
assessments proved to be useful vehicles for evaluating the extent to which GALAXY students 
understood die core science concepts presented in each of the themes. Though there did appear 
to be some age-related differences in understanding these concepts, the majority of GALAXY 
students across all grades demonstrated that they understood these "big ideas" and often were able to 
apply them in new contexts. Further, when non-GALAXY comparison students were evaluated on two 
of the four assessments, GALAXY students outperformed them in almost every aspect. 
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V. Changes in attitudes 



The philosophy of GALAXY Classroom Science has been expressed as "science is the 
investigation of phenomena by exploring about, asking questions, inventing ideas, and loving 
it!" Teachers and students who participated in the demonstration phase of GALAXY science 
spanned a continuum from being uncertain about whether they liked science to being strong 
supporters of hands-on science. Evaluators were interested in what changes in attitudes among 
both teachers and students might be produced by participation in GALAXY science. 

Teacher Attitudes 

FWL researchers surveyed teachers' attitudes towards teaching science both before and after 
they taught GALAXY Classroom Science for Grades 3-5. Ratings ranged from 1 (strongly 
disagree) to 5 (strongly agree) on items such as comfort in teaching science, hands-on 
exploration by students, and availability of appropriate materials. 

The scores of GALAXY teachers on the pre-GALAXY survey were quite high, indicating that 
they already supported hands-on science and were fairly coiifident about their ability to teach 
it. For example, teachers very strongly agreed (4.5 out of 5) that elementary students should 
participate in hands-on science and that boys and girls were both interested in science. 

Given these initially high levels of appropriate attitudes and practices, it was somewhat 
surprising to find statistically significant changes among GALAXY teachers in a number of key 
areas (5 out of 24 attitude items and 3 out of 9 teadung practices). Teachers moved from 
comfortable to very comfortable about teaching science (from a mean of 3.79 to 4.52, p<.001) 
and from calling dieir materials slightly inadequate (2.78) to quite adequate (4.12, p<.001). 
They also indicated that they were better prepared to teach science (3.26 to 4.15, p<.001). 
They even reported an increase in favorable parent comment about students enjoying science 
(3.29 to 3.86). Statistically significant gains also were reported in the frequency with which 
GALAXY teachers encouraged students to write about science, used science to teach math, and 
had hands-on science activities in class. In addition, the reported frequency of science teaching 
increased. 

When teachers were asked on the end-of-year survey how GALAXY had affected their teaching, 
a number of them gave responses that indicated that their teaching, not just their science 
teaching, had been permanently changed: 

"It's changed my philosophy and how I plan for teaching both in terms of goals and 
how I interact in class: asking questions, probing far better answers, striving for 
excellence." 

— GALAXY teacher from Texas 

"I will never be textbook bound again; even in third grade, children still need the 
hands-on experience to explore concepts before higher level thinking can consistently 
occur." 

— GALAXY teacher from South Carolina 

"I have done a better job at accepting all ideas and seeing how some strange comments 
do fit. I have done a 100% better job teaching science. I have lost some of my fears 
about experiments. I have tried more group projects in other subject areas." 

— GALA>^ teacher from Colorado 

"[As a result of GALAXY,] I've also changed to using more cooperative activities 
throughout the curriculum (math, social studies, etc.). I feel this has made me a more 
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effective teacher, and my kids are having fun." 

— GALAXY teacher from California 

"GALAXY has shown me that my role in the classroom should be more as facilitator' 
and less as 'lecturer. ' I have also learned how much more children enjoy learning when 
it is learner-driven and they have control over what they learn and how they learn it. " 

— GALAXY teacher from Maryland 

Participation in GALAXY Classroom Science facilitated a very positive set of outcomes among 
the teachers. Hands-on science requires an extra commitment of both time and intellectual 
engagement by teachers, factors that may deter many from doing it. The evaluation data 
indicates that being a part of the GALAXY Classroom overcame these impediments and 
brought exciting science teaching into the lives of teachers and students throughout the country. 

Student Attitudes 

FWL evaluators developed a 27-item survey to measure student attitudes to science and 
administered it to both GALAXY and comparison students before and after GALAXY 
Classroom Science. As part of the analysis, the 27 attitude items were clustered into groups 
based on their intercorrelations and a cluster score was computed. 

There was empirical evidence for four clusters of items, and each was given a descriptive name 
for convenience of reference: "fun," "learning," "mystique," and "world" clusters. TTie changes 
associated with these four clusters are shown in Figure 8. 
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Figure 8. Mean Attitude Change by Cluster for GALAXY and Comparison Students 

For a group of nine items, labeled the "fun cluster," GALAXY students had a more positive 
attitude than comparison students towards how much fun science was at the pre-test, and that 
gap widened significantly by the time of the post-test (see Figure 8). GALAXY students 
showed a statistically significant improvement in their perspective on five out of nine items 
related to doing science at school, whereas comparison students only had such changes on two 
items. This evidence suggests that participating in GALAXY Classroom Science positively changes 
student attitudes about science class. 
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Another cluster was a "learning" duster consisting of four items. Overall, the "learning" cluster 
score started with GALAXY students significantly more positive and ended with the gap even 
larger, but the difference between the two groups in the amount of increase was not statistically 
significant. At the post-test, GALAXY students had higher mean scores than the comparison 
students by .10 to .13 for all four individual items, a consistent and substantial margin. The 
evidence from this cluster suggests that participating in GALAXY science classes had a 
beneficial effect on students' attitudes to engaging in activities in class and not being afraid to 
make mistakes. GALAXY students are not afraid to "mess around," one of GALAXY'S goals. 

Five more items comprise a cluster that relates to the "mystique" of science. The changes on the 
"mystique" cluster score and on the negatively-phrased items in this cluster indicate that 
science had lost much of its mystique for both GALAXY and comparison students, with a 
significantly greater decrease for GALAXY students. All of the significant differences for this 
cluster of attitude items favor GALAXY over comparison students. 

The fourth cluster, the "world" cluster, is composed of nine items that generally relate to the 
relationship between science and the world outside school. This cluster provides some 
counterpoint to the strongly favorable results from the other three clusters of student attitudes. 
For these items, GALAXY students not only do not have more positive attitudes than the 
comparison students at post-test, their decrease in agreement is significant and significantly 
greater than the change for comparison students (who actually improved slightly). The main 
exception is an item diat said "Science teaches us to try out new ideas," on which GALAXY 
students increased their agreement markedly more than comparison students, to extraordinarily 
high levels. 

The evidence from this "world" cluster is unexpected and suggests that GALAXY students are 
having difficulty making the connection between the fun things they do in science class and the 
way science is used in the reed world. Perhaps the GALAXY Classroom Science curriculum 
needs to be strengthened in areas making the link between the classroom and the larger society. 

The results of the student attitude survey confirm that participation in GALAXY Classroom Science 
positively affected the attitudes of students towards participating in science class, engaging in activities 
in which they did not know the right answer, and feeling comfortable with GALAXY -promoted 
learning strategies. But the attitude survey results also suggest that the connection between what was 
happening in the classroom and the larger world might have eluded them. This is an area that can be 
added to the curriculum, now that GALAXY has been successful in getting students to like 
science class. 
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VI. Conclusion 



"I have been absolutely thrilled with this science program. It has been an incredible 
experience to watch my children become 'scientists' in the classroom. I have not used 
any program as well-integrated or as motivating to all my students as the GALAXY 
science program." 

— GALAXY teacher from Oregon 

One of galaxy's primary objectives has been to get students and teachers excited about 
science and to motivate them to jump in, begin exploring, keep asking questions, and have fun. 
As the evaluation evidence from both teachers and students indicates, GALAXY Classroom 
Science is a highly successful initiative. 

GALAXY'S broadcasts motivate student interest and provide a realistic context for the hands- 
on science activities. The thematic structure of the curriculum helps to integrate a wide variety 
of experiences: participating in hands-on inquiry, writing and communicating through the fax, 
and doing take-home activities. The teacher institutes and ongoing teacher support have 
assisted teachers in weaving these elements together. 

Although teachers are still learning how to use GALAXY science optimally in their classrooms, 
the evaluation results point to positive experiences in most classrooms: science is being taught 
regularly, students are excitedly engaged in hands-on activities, and teachers are developing a 
new en^usiasm for science and science teaching. 

The preceding sections of this report describe the effects on students and teachers of the various 
curricular, technological, and staff development components of GALAXY science in schools 
throughout the coimtry. As this report indicates, it is certainly the case with GALAXY science 
that the whole is greater than the sum of its parts. The components by themselves represent 
exemplary efforts in science education. Together, they create a powerful package. 
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